{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第4周-聚类、降维、矩阵分解 \n",
    "问题描述\n",
    "根据活动的关键词（count_1, count_2, ..., count_100，count_other属性）做聚类，可采用KMeans聚类\n",
    "尝试K=10，20，30，..., 100, 并计算各自CH_scores。\n",
    "解题提示\n",
    "文件说明：\n",
    "1. 可以先运行0. EDA.ipynb，看一下竞赛所有数据的情况；\n",
    "2. 总体活动的数目太多（300w+记录），可以只需对训练集train.csv和测试集test.cv出现的活动（13418条记录）举行聚类即可。运行1. Users_Events.ipynb可得到只在训练集train.csv和测试集test.cv出现的活动，可自己修改代码存为csv格式，在进行聚类。\n",
    "批改标准\n",
    "1. 抽取出只在训练集和测试集中出现的event：20分\n",
    "2. 聚类 ：40分\n",
    "3. CH_scores计算：20分\n",
    "4. 结果显示/分析：20分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  根据训练集和测试集上的活动id 找出对应于event.csv的相关信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### （1） 导入数据包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#保存数据\n",
    "import cPickle\n",
    "\n",
    "import itertools\n",
    "\n",
    "#处理事件字符串\n",
    "import datetime\n",
    "\n",
    "import numpy as np\n",
    "import scipy.io as sio\n",
    "import scipy.sparse as ss\n",
    "\n",
    "#相似度/距离\n",
    "import scipy.spatial.distance as ssd\n",
    "\n",
    "from collections import defaultdict\n",
    "from sklearn.preprocessing import normalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### （2） 从测试集和训练集中找到相应的不重复的eventID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 统计训练集中有多少不同的用户的events\n",
    "uniqueEvents = set()\n",
    "\n",
    "for filename in [\"train.csv\", \"test.csv\"]:\n",
    "    f = open(filename, 'rb')\n",
    "    \n",
    "    #忽略第一行（列名字）\n",
    "    f.readline().strip().split(\",\")\n",
    "    \n",
    "    for line in f:    #对每条记录\n",
    "        cols = line.strip().split(\",\")\n",
    "        uniqueEvents.add(cols[1])   #第二列为活动ID\n",
    "    f.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13418"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(uniqueEvents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### （3） 根据不重复的eventID 找到对应 event.csv 中的信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 打开输出文件\n",
    "fo = open(\"homework_event.csv\", \"w\")\n",
    "\n",
    "# 读入活动文件\n",
    "fi = open(\"events.csv\", 'rb')\n",
    "\n",
    "# 保存表头\n",
    "fo.write(fi.readline())\n",
    "\n",
    "# 保存对应event 的记录\n",
    "for line in fi:    \n",
    "    cols = line.strip().split(\",\")\n",
    "    \n",
    "    if (cols[0] in uniqueEvents):\n",
    "        fo.write(line)\n",
    "        \n",
    "fi.close()\n",
    "fo.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
